Image super-resolution method, image super-resolution device, and computer readable storage medium

ABSTRACT

Disclosed is an image super-resolution method, which includes: acquiring and amplifying an image to be processed, and extracting a scaling feature from the amplified image, to obtain a first image to be processed; sending the first image to be processed to a residual network, for the residual network outputting a corrected second image to be processed; and restoring the second image to be processed to generate a restored image, and outputting the restored image. The present disclosure further provides an image super-resolution device and a computer readable storage medium.

CROSS REFERENCE TO RELATED APPLICATIONS

The present disclosure is a Continuation Application of PCT Applicationwith No. PCT/CN2018/099071, filed on Aug. 7, 2018, which claims thepriority of Chinese Patent Application with No. 201810147781.5, entitled“image super-resolution method, image super-resolution device, andcomputer readable storage medium”, filed on Feb. 11, 2018, which ishereby incorporated by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to the technical field of imageprocessing, and in particular, to an image super-resolution method, animage super-resolution device, and a computer readable storage medium.

BACKGROUND

Nowadays, high-resolution display screens has been widely used in homeaudio and video equipments and mobile devices, as people are ofincreasingly demanding of high-quality images or videos. High-qualityimages or videos help people to obtain more abundant and accurateinformation, and resolution is an important factor for judging thequality of images. However, in many cases, the resolution of thecaptured image is lower due to the limitation of the performances ofexternal machines and the influences of shooting conditions, the blurredimage cannot meet the requirements of practical application. Therefore,it is imperative to improve the image resolution. Although the mostdirect way to improve the image resolution is to improve the hardwareconfiguration, the image resolution which is also called assuper-resolution technology is usually improved through the softwaremethod, as improving the hardware configuration is costly and limited bythe physical condition.

Super-resolution image reconstruction can generate high-quality andhigh-resolution images through a set of low-quality and low-resolutionimages or a motion sequence. With the development of artificialintelligence, technicians can get super-resolution single frame based ondepth convolution neural network, making great progress insuper-resolution technology of single frame.

At present, super-resolution image reconstruction has been widely usedin real life, such as, high-definition television, medical images,satellite images, security detection, microscopic imaging, virtualreality, etc. As to the field of digital television, super-resolutionimage reconstruction which can convert digital television signals intohigh-definition television signals is an extremely importantapplication, for effectively improving video clarity. Super-resolutionimage reconstruction follows the principle of “deeper network, bettereffect”. However, due to the deepening of the network, technologyadopting SRResNet network structure has defects such as too manyparameters, slow gradient convergence, difficult training, a decrease inreal-time rate, etc. The classical ResNet model uses batch normalizationmethod to converge the gradient and speed up the training process.However, batch normalization may lead to excessive computationaloverhead with the deepening of the depth of the network, and thefeatures may be standardized in the light of its principle. As such,batch normalization is not suitable for super-resolution applications.Therefore, it is necessary to propose a processing method different frombatch normalization to reduce computational overhead and speed up theconvergence rate. Additionally, the classic ResNet model does notmention how to realize super-resolution technology with differentmagnification rates. Since the magnification rate of the resolution inTV applications is fixed, the classic ResNet model cannot be welladapted to TV applications.

The contents above are only intended to assist in understanding thetechnical solution of the present disclosure, but not to represent therelated art.

SUMMARY OF THE DISCLOSURE

It is therefore one main objective of the disclosure to provide an imagesuper-resolution method, an image super-resolution device, and acomputer readable storage medium, aiming to solve the problem that videoimages at different magnification rates cannot share training results ofconvolutional neural network

In order to achieve the above objective, the present disclosure providesan image super-resolution method, which includes:

acquiring and amplifying an image to be processed, and extracting ascaling feature from the amplified image, to obtain a first image to beprocessed;

sending the first image to be processed to a residual network, for theresidual network outputting a corrected second image to be processed;and

restoring the second image to be processed to generate a restored image,and outputting the restored image.

Optionally, the operation of acquiring and amplifying an image to beprocessed, and extracting a scaling feature from the amplified image, toobtain a first image to be processed, includes:

acquiring a low-resolution image to be processed, and pre-processing theimage to be processed in a pre-processing convolution layer; and

sending the pre-processed image to be processed to a scale amplificationmodule, amplifying the image to be processed based on a presetamplification scale, and extracting the scaling feature from theamplified image, to obtain the first image to be processed.

Optionally, the preset amplification scale is defined as two times,three times, or four times.

Optionally, the operation of sending the first image to be processed toa residual network, for the residual network outputting a correctedsecond image to be processed, includes:

sending the first image to be processed to the residual network,processing the first image to be processed by a plurality of bottleneckresidual units in the residual network, to generate the corrected secondimage to be processed; and

sending the second image to be processed to a scale restoring module.

Optionally, the residual network includes the plurality of bottleneckresidual units and a convolution layer, and each bottleneck residualunit is connected with a weight normalization module.

Optionally, the bottleneck residual unit includes three convolutionlayers, and an activation function layer is defined between each twoadjacent convolution layers, and the activation function is a PReLufunction.

Optionally, the activation function includes a variable, and the valueof the variable is obtained through learning from an upper networklayer.

Optionally, the operation of restoring the second image to be processedto generate a restored image, and outputting the restored image,includes:

on condition that a scale restoring module receives the second image tobe processed, reducing the scale of the second image to be processedbased on a scale in a scale amplification module, to generate therestored image; and

outputting the restored image.

Optionally, the operation of sending the first image to be processed toa residual network, for the residual network outputting a correctedsecond image to be processed, includes:

sending the first image to be processed to the residual network,processing the first image to be processed by a plurality of bottleneckresidual units in the residual network, to generate the corrected secondimage to be processed; and

sending the second image to be processed to a scale restoring module.

In addition, in order to achieve the above objective, the presentdisclosure further provides an image super-resolution device. The deviceincludes: a memory, a processor, and an image super-resolution programstored on the memory and executable on the processor, the program, whenexecuted by the processor, implements the following operations:

acquiring and amplifying an image to be processed, and extracting ascaling feature from the amplified image, to obtain a first image to beprocessed;

sending the first image to be processed to a residual network, for theresidual network outputting a corrected second image to be processed;and

restoring the second image to be processed to generate a restored image,and outputting the restored image.

Optionally, the program, when executed by the processor, implements thefollowing operations:

acquiring a low-resolution image to be processed, and pre-processing theimage to be processed in a pre-processing convolution layer; and

sending the pre-processed image to be processed to a scale amplificationmodule, amplifying the image to be processed based on a presetamplification scale, and extracting the scaling feature from theamplified image, to obtain the first image to be processed.

Optionally, the program, when executed by the processor, implements thefollowing operations:

sending the first image to be processed to the residual network,processing the first image to be processed by a plurality of bottleneckresidual units in the residual network, to generate the corrected secondimage to be processed; and

sending the second image to be processed to a scale restoring module.

In addition, in order to achieve the above objective, the presentdisclosure further provides a computer readable storage medium, an imagesuper-resolution program is stored on the computer readable storagemedium, the program, when executed by the processor, implements thefollowing operations:

acquiring and amplifying an image to be processed, and extracting ascaling feature from the amplified image, to obtain a first image to beprocessed;

sending the first image to be processed to a residual network, for theresidual network outputting a corrected second image to be processed;and

restoring the second image to be processed to generate a restored image,and outputting the restored image.

Optionally, the program, when executed by the processor, implements thefollowing operations:

acquiring a low-resolution image to be processed, and pre-processing theimage to be processed in a pre-processing convolution layer; and

sending the pre-processed image to be processed to a scale amplificationmodule, amplifying the image to be processed based on a presetamplification scale, and extracting the scaling feature from theamplified image, to obtain the first image to be processed.

Optionally, the program, when executed by the processor, implements thefollowing operations:

sending the first image to be processed to the residual network,processing the first image to be processed by a plurality of bottleneckresidual units in the residual network, to generate the corrected secondimage to be processed; and

sending the second image to be processed to a scale restoring module.

In the technical solution of the present disclosure, the image to beprocessed is acquired and amplified, and the scaling feature isextracted from the amplified image, to obtain the first image to beprocessed; the first image to be processed is sent to the residualnetwork, for the residual network outputting the corrected second imageto be processed; and the second image to be processed is restored togenerate the restored image, and the restored image is output. Thepresent disclosure could pre-process the image by amplifying the imageto different magnifications. The module part relying on magnificationrates can be separated from the main network, and most of the parametersindependent of the magnification rates can share the network trainingresults at different magnification rates, for increasing theversatility. The technical solutions can be meet the super-resolutionrequirement of 8K TV (the total column quantity of video pixels is4320).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic structural diagram of a terminal of an imagesuper-resolution device in a hardware operating environment according toan embodiment of the present disclosure;

FIG. 2 is a flow chart of an image super-resolution method according toa first embodiment of the present disclosure;

FIG. 3 is a main flow chart of the image super-resolution methodaccording to the first embodiment of the present disclosure;

FIG. 4 is a schematic structural diagram of a bottleneck residual unitin the image super-resolution method according to the first embodimentof the present disclosure;

FIG. 5 is a detailed flow chart of the operations of acquiring andamplifying an image to be processed, and extracting a scaling featurefrom the amplified image, to obtain a first image to be processed, in animage super-resolution method according to a second embodiment of thepresent disclosure;

FIG. 6 is a detailed flow chart of the operation of sending the firstimage to be processed to a residual network, for the residual networkoutputting a corrected second image to be processed, in an imagesuper-resolution method according to a third embodiment of the presentdisclosure;

FIG. 7 is a detailed flow chart of the operations of restoring thesecond image to be processed to generate a restored image, andoutputting the restored image, in an image super-resolution methodaccording to a fourth embodiment of the present disclosure.

The realization of the aim, functional characteristics, advantages ofthe present disclosure are further described specifically with referenceto the accompanying drawings and embodiments.

DETAILED DESCRIPTION OF THE EMBODIMENTS

It is to be understood that, the exemplary embodiments of the presentdisclosure are configured for illustrating the present disclosure ratherthan restricting the present disclosure.

As shown in FIG. 1, FIG. 1 shows a schematic structural diagram of aterminal of an image super-resolution device in a hardware operatingenvironment according to an embodiment of the present disclosure.

In the embodiment of the present disclosure, the terminal may be a PC,or a mobile terminal device with display function, such as a smartphone, a tablet computer, an e-book reader, a Moving Picture ExpertsGroup Audio Layer III (MP3) player, a Moving Picture Experts Group AudioLayer IV (MP4) player, a portable computer, etc.

As shown in FIG. 1, the terminal may include a processor 1001, such as aCPU, a network interface 1004, a user interface 1003, a memory 1005, anda communication bus 1002. The communication bus 1002 is configured torealize the connections and communications among these components. Theuser interface 1003 may include a display, an input unit such as aKeyboard. Optionally, the user interface 1003 may also include astandard wired interface and a wireless interface. The network interface1004 may optionally include a standard wired interface and a wirelessinterface (such as a WI-FI interface). The memory 1005 may be ahigh-speed RAM memory or a non-volatile memory such as a disk memory.The memory 1005 may alternatively be a storage device independent of theaforementioned processor 1001.

Optionally, the terminal may further include a camera, a Radio Frequency(RF) circuitry, a sensor, an audio circuitry, a WiFi module, etc. Thesensor can be at least one selected from a group consisting of lightsensor, motion sensor, etc. Specifically, the light sensor may includean ambient light sensor and a proximity sensor, and the ambient lightsensor may adjust brightness of the display screen according tobrightness of ambient light, and the proximity sensor may turn off thedisplay screen and/or backlight when the mobile terminal moves to theear. The gravity acceleration sensor, as a kind of motion sensor, candetect the magnitude of acceleration in all directions (generallyincluding x axis, y axis, z axis), and can detect the magnitude anddirection of gravity when the mobile terminal is still. So that, thegravity acceleration sensor can be applied to applications foridentifying attitude of mobile terminal (such as switching betweenhorizontal orientation and vertical orientation, related games,magnetometer attitude calibration), and can be also applied to functionsrelated to vibration identification (such as pedometer, tapping), etc.Of course, the mobile terminal can also be equipped with gyroscope,barometer, hygrometer, thermometer, infrared sensor, and the like, thereis no need to repeat here.

Those skilled in the art can understand that the structure shown in FIG.1 does not constitute a limitation on the terminal, and the terminal mayinclude more or fewer components than shown, or a combination of some ofthe components, or different component arrangements.

As shown in FIG. 1, the memory 1005 which is regarded as a computerstorage medium may include an operating system, a network communicationmodule, a user interface module, and an image super-resolution program.

In the terminal as shown in FIG. 1, the network interface 1004 is mainlyconfigured to connect with a background server, and perform datacommunication with the background server. The user interface 1003 ismainly configured to connect with the client (a user end), and performdata communication with the client. While the processor 1001 may beconfigured to call an image super-resolution program stored on thememory 1005.

In this embodiment, the image super-resolution device includes a memory1005, a processor 1001, and an image super-resolution program stored onthe memory 1005 and executable on the processor 1001. The program, whenexecuted by the processor 1001, performs the following operations:

acquiring and amplifying an image to be processed, and extracting ascaling feature from the amplified image, to obtain a first image to beprocessed;

sending the first image to be processed to a residual network, for theresidual network outputting a corrected second image to be processed;and

restoring the second image to be processed to generate a restored image,and outputting the restored image.

The program, when executed by the processor, further implements thefollowing operations:

acquiring a low-resolution image to be processed, and pre-processing theimage to be processed in a pre-processing convolution layer; and

sending the pre-processed image to be processed to a scale amplificationmodule, amplifying the image to be processed based on a presetamplification scale, and extracting the scaling feature from theamplified image, to obtain the first image to be processed.

The program, when executed by the processor, further implements thefollowing operations:

sending the first image to be processed to the residual network,processing the first image to be processed by a plurality of bottleneckresidual units in the residual network, to generate the corrected secondimage to be processed; and

sending the second image to be processed to a scale restoring module.

The present disclosure provides an image super-resolution methodaccording to a first embodiment. As shown in FIG. 2, FIG. 2 shows a flowchart of the image super-resolution method according to the firstembodiment of the present disclosure;

S10, acquiring and amplifying an image to be processed, and extracting ascaling feature from the amplified image, to obtain a first image to beprocessed;

The method of the present disclosure can be applied to both the imagefield and the video field. In the method, single frame is acquiredfirst. As to the video, the video can be decomposed into a sequencehaving continuous frames, and then the frames in the sequence aresubjected to a super-resolution processing, and the processed frames areintegrated into high-resolution video based on the processedhigh-resolution avatars.

FIG. 3 shows the main flow chart of the method. When a low-resolutionimage to be processed is obtained, the image to be processed is inputinto the pre-processing convolution layer to extract feature from theimage to be processed. After the feature extracting process, the imagesubjected to feature extracting process is sent to the scaleamplification module. Three amplification scales are preset in the scaleamplification module, namely two times, three times and four timesrespectively, so that different amplification scales can be selectedaccording to actual conditions. Since the video resolution adopted bydigital TV is fixed, such as 720P, 1080P, 2K, or 4K, etc. Under normalcircumstances, different video resolutions are suitable for differentamplification scales. When the amplification scale is two times, thescale of each pixel point in a single direction changes to two times ofthe original scale, that is, each pixel point changes to four pixelpoints. And the four pixel points are arranged in 2×2 array. That is,the scale of the amplified pixel point in any direction becomes twotimes of the original scale of the pixel point. Similarly, when theamplification scale is three times or four times, the scale of eachpixel point changes to three times or four times of the original scaleof the pixel point. For example, when the amplification scale is threetimes, one pixel point changes to nine pixel points; when theamplification scale is four times, one pixel point changes to sixteenpixel points.

After the scale of image to be processed is subjected to theamplification process by the scale amplification module, the scalingfeature is extracted from the amplified image. Scaling featurerepresents the amplification times which indicates the amplification ofimage. Then, the first image to be processed is acquired after processof extracting scaling feature. The first image to be processed does notcontain the scaling feature. That is, even the images are amplifiedbased on different amplification scales, the first imaged to beprocessed, obtained through the process of extracting scaling feature,are the same.

after scaling feature extraction is the same.

S20, sending the first image to be processed to a residual network, forthe residual network outputting a corrected second image to beprocessed; and

Actually, the residual network is a convolution neural network, theresidual network includes several bottleneck residual units andconvolution layers. A weight normalization module is set after eachbottleneck residual unit. As shown in FIG. 4, each bottleneck residualunit includes three convolution layers and two activation functionlayers, and one activation function layer is located between twoadjacent convolution layers. The activation function adopts the PReLufunction and contains a variable, the variable is obtained from thenetwork learning at the previous level. In addition, in this embodiment,ResNet-34 network model is adopted.

Before the emergence of residual network, depth network model adapted byuser has a few layers. A series of measures, such as setting reasonableweight initialization, increasing batch standardization, and improvingactivation function, etc., were adopted to effectively alleviate thegradient disappearance, making depth network training feasible. With thedeepening of network layers, the error becomes smaller theoretically,and the expression ability of the model is enhanced. However, thetraining error becomes larger after simply overlaying the networklayers, which is mainly affected by factors such as gradientdisappearance, etc. Thus, the residual network is emerged. The residualnetwork is formed by stacking residual modules. The residual moduleincludes a conventional residual module and a bottleneck residualmodule. The 1×1 convolution in the bottleneck residual module can playthe role of lifting and lowering the dimension, thus enabling the 3×3convolution can be performed on the lower input dimension. This designcan greatly reduce the calculation amount, and can achieve a bettereffect in deep network. The activation function in the residual networkis defined as PReLu, instead of ReLu, and a parameter is introduced tohelp the activation function to learn partial negative coefficientadaptively. In addition, the residual network adopts image up-samplingmethod and sub-pixel convolution layer.

S30, restoring the second image to be processed to generate a restoredimage, and outputting the restored image.

A scale restoring module is arranged behind the residual network, andthe main function of the scale restoring module is to reductivelyrestore the image to be processed which is amplified by the scaleamplification module, for generating the high-resolution restored image,then the high-resolution restored image is output, for obtaininghigh-quality video.

In the technical solution of the present disclosure, the image to beprocessed is acquired and amplified, and the scaling feature isextracted from the amplified image, to obtain the first image to beprocessed; the first image to be processed is sent to the residualnetwork, for the residual network outputting the corrected second imageto be processed; and the second image to be processed is restored togenerate the restored image, and the restored image is output. Thepresent disclosure could pre-process the image by amplifying the imageto different magnifications. The module part relying on magnificationrates can be separated from the main network, and most of the parametersindependent of the magnification rates can share the network trainingresults at different magnification rates, for increasing theversatility. The technical solutions can be meet the super-resolutionrequirement of 8K TV (the total column quantity of video pixels is4320).

Based on the first embodiment, the present disclosure provides an imagesuper-resolution method according to a second embodiment. Referring toFIG. 5, S10 includes:

S11, acquiring a low-resolution image to be processed, andpre-processing the image to be processed in a pre-processing convolutionlayer; and

When the low-resolution image to be processed is acquired, the image tobe processed is input into the pre-processing convolution layer toextract feature from the image to be processed. After the featureextracting process, the image subjected to feature extracting process issent to the scale amplification module.

S12, sending the pre-processed image to be processed to a scaleamplification module, amplifying the image to be processed based on apreset amplification scale, and extracting the scaling feature from theamplified image, to obtain the first image to be processed.

Three amplification scales are preset in the scale amplification module,namely two times, three times and four times respectively, so thatdifferent amplification scales can be selected according to actualconditions. Since the video resolution adopted by digital TV is fixed,such as 720P, 1080P, 2K, or 4K, etc. Under normal circumstances,different video resolutions are suitable for different amplificationscales. After the scale of image to be processed is subjected to theamplification process by the scale amplification module, the scalingfeature is extracted from the amplified image. Scaling featurerepresents the amplification times which indicates the amplification ofimage. Then, the first image to be processed is acquired after processof extracting scaling feature. The first image to be processed does notcontain the scaling feature. That is, even the images are amplifiedbased on different amplification scales, the first imaged to beprocessed, obtained through the process of extracting scaling feature,are the same.

Further, in one embodiment, the method further includes:

The preset amplification scale is defined as two times, three times, orfour times.

When the amplification scale is defined as two times, the scale of eachpixel point in a single direction changes to two times of the originalscale, that is, each pixel point changes to four pixel points. And thefour pixel points are arranged in 2×2 array. That is, the scale of theamplified pixel point in any direction becomes two times of the originalscale of the pixel point. Similarly, when the amplification scale isthree times or four times, the scale of each pixel point changes tothree times or four times of the original scale of the pixel point. Forexample, when the amplification scale is three times, one pixel pointchanges to nine pixel points; when the amplification scale is fourtimes, one pixel point changes to sixteen pixel points.

In the embodiment, the image super-resolution method obtains thelow-resolution image to be processed, and pre-processes the image to beprocessed in the pre-processing convolution layer, then sends thepre-processed image to the scale amplification module for amplifying thepre-processed image based on the preset amplification scale, andextracts the scaling feature from the amplified image to obtain thefirst image to be processed which is subjected to scaling featureextracting process. The adjacent frames of the video images have strongcorrelation, not only the quality of reconstructed super-resolutionimages, but also the efficiency of reconstructing super-resolutionimages should be ensured.

Based on the first embodiment, the present disclosure provides an imagesuper-resolution method according to a third embodiment. Referring toFIG. 6, S20 includes:

S21, sending the first image to be processed to the residual network,processing the first image to be processed by a plurality of bottleneckresidual units in the residual network, to generate the corrected secondimage to be processed; and

Before the emergence of residual network, depth network model adapted byuser has a few layers. A series of measures, such as setting reasonableweight initialization, increasing batch standardization, and improvingactivation function, etc., were adopted to effectively alleviate thegradient disappearance, making depth network training feasible. With thedeepening of network layers, the error becomes smaller theoretically,and the expression ability of the model is enhanced. However, thetraining error becomes larger after simply overlaying the networklayers, which is mainly affected by factors such as gradientdisappearance, etc. Thus, the residual network is emerged. The residualnetwork is formed by stacking residual modules. The residual moduleincludes a conventional residual module and a bottleneck residualmodule. The 1×1 convolution in the bottleneck residual module can playthe role of lifting and lowering the dimension, thus enabling the 3×3convolution can be performed on the lower input dimension. This designcan greatly reduce the calculation amount, and can achieve a bettereffect in deep network. The activation function in the residual networkis defined as PReLu, instead of ReLu, and a parameter is introduced tohelp the activation function to learn partial negative coefficientadaptively. In addition, the residual network adopts image up-samplingmethod and sub-pixel convolution layer.

Actually, the residual network is a convolution neural network, theresidual network includes several bottleneck residual units andconvolution layers. A weight normalization module is set after eachbottleneck residual unit. As shown in FIG. 4, each bottleneck residualunit includes three convolution layers and two activation functionlayers, and one activation function layer is located between twoadjacent convolution layers. The activation function adopts the PReLufunction and contains a variable, the variable is obtained from thenetwork learning at the previous level. In addition, in this embodiment,ResNet-34 network model is adopted.

S22, sending the second image to be processed to a scale restoringmodule.

The first image to be processed which does not have the scaling featureis input into the residual network, for processing the first image to beprocessed, then the the residual network outputs the corrected secondimage to the scale restoring module, the scale restoring module restoresthe image which does not have the scaling feature, for generating therestored image with scaling feature.

Furthermore, in one embodiment, the method further includes:

The residual network includes the plurality of bottleneck residual unitsand a convolution layer, and each bottleneck residual unit is connectedwith a weight normalization module.

As shown in FIG. 3, the residual network is arranged behind the scaleamplification module and includes a plurality of bottleneck residualunits. A convolution layer is arranged behind the bottleneck residualunit. A weight normalization module is arranged behind each bottleneckresidual unit. The weight normalization processing is a method forparameterizing a neural network model. Since the parameter set includesa vast number of weights and deviation values in the depth neuralnetwork, how to optimize the above-mentioned parameters is an importantproblem in depth learning.

In the weight normalization module, in order to speed up the convergencespeed of the optimization operation, the K-order weight vector isexpressed by the K-order vector v and the scale factor g based on therandom gradient descent. Through certain mathematical changes, thefollowing formula is obtained:

${w = {\frac{g}{v}v}},{{w} = g}$${\nabla_{g}L} = \frac{{\nabla_{w}L} \cdot v}{v}$${\nabla_{v}L} = {{\frac{g}{v}{\nabla_{w}L}} - {\frac{g\; {\nabla_{g}L}}{{v}^{2}}v}}$

Where g is regarded as the scale factor, w is regarded as the k-orderweight vector, v is regarded as the k-order vector, and L is regarded asthe loss function.

Furthermore, in one embodiment, the method further includes:

The the bottleneck residual unit includes three convolution layers, andan activation function layer is defined between each two adjacentconvolution layers, and the activation function is a PReLu function.

As shown in FIG. 4, each bottleneck residual unit includes threeconvolution layers, and an activation function layer is defined betweeneach two adjacent convolution layers. The activation function isparametric ReLU, i.e., PReLU function. The formula for this function isas follows:

${PReLu} = \{ \begin{matrix}{x,{x \geq 0}} \\{{\alpha \cdot x},{x < 0}}\end{matrix} $

In which, α is a variable and learned from the upper layer of network.The variable α helps the function to learn some negative coefficientsadaptively.

As to the image super-resolution method according to the embodiment, thefirst image to be processed is sent to the residual network, and thenprocessed by the plurality of bottleneck residual units in the residualnetwork, for generating corrected second image to be processed. Then thecorrected second image is sent to the scale restoring module. Therefore,the activation function layer is improved, for improving the learningability and adaptability of the residual network.

Based on the first embodiment, the present disclosure provides an imagesuper-resolution method according to a fourth embodiment. Referring toFIG. 7, S30 includes:

S31, on condition that a scale restoring module receives the secondimage to be processed, reducing the scale of the second image to beprocessed based on a scale in a scale amplification module, to generatethe restored image; and

A scale restoring module is arranged behind the residual network, andthe main function of the scale restoring module is to reductivelyrestore the image to be processed which is amplified by the scaleamplification module, for generating the high-resolution restored image,then the high-resolution restored image is output, for obtaininghigh-quality video.

S32, outputting the restored image.

As to the image super-resolution method in the embodiment, on conditionthat the scale restoring module receives the second image to beprocessed, the scale restoring module generates the restored image bycorrespondingly reducing the second image based on the scale in thescale amplification module. Then the stored image is output. Theresidual network adopts the weight normalization module for greatlyreducing the calculation cost of weight normalization, avoids the add ofrandomness in the noise estimation process. Therefore, the residualnetwork can be applied to more types of network models.

In addition, the present disclosure also provides a computer readablestorage medium. The computer readable storage medium stores an imagesuper-resolution program. The program, when executed by a processor,implements the following operations:

acquiring and amplifying an image to be processed, and extracting ascaling feature from the amplified image, to obtain a first image to beprocessed;

sending the first image to be processed to a residual network, for theresidual network outputting a corrected second image to be processed;and

restoring the second image to be processed to generate a restored image,and outputting the restored image.

The program, when executed by the processor, further implements thefollowing operations:

acquiring a low-resolution image to be processed, and pre-processing theimage to be processed in a pre-processing convolution layer; and

sending the pre-processed image to be processed to a scale amplificationmodule, amplifying the image to be processed based on a presetamplification scale, and extracting the scaling feature from theamplified image, to obtain the first image to be processed.

The program, when executed by the processor, implements the followingoperations:

sending the first image to be processed to the residual network,processing the first image to be processed by a plurality of bottleneckresidual units in the residual network, to generate the corrected secondimage to be processed; and

sending the second image to be processed to a scale restoring module.

It needs to be noted that in the present disclosure, the terms“comprising”, “including” or other variants aim to cover non-exclusiveinclusion, such that the processes, methods, articles or devicesincluding a series of factors not only include these factors, but alsoinclude other factors not listed explicitly, or further comprise includeintrinsic for such processes, methods, articles or devices. In theabsence of more limitations, the factors limited by “comprising a . . .” do not exclude that additional identical factors are also included inthe processes, methods, articles or devices comprising said factors.

The sequence number in the above embodiments of the present disclosureis only for the purpose of explanation and not intended to indicate themerits of the embodiments.

Through above description of the embodiments, it should be understood bya person skilled in the art that the present disclosure may beimplemented by means of software in connection with necessary universalhardware platform. Of course, the present disclosure may also beimplemented by a hardware. However, in many cases the former is morepreferred. Based on this understanding, all or the part contributing tothe prior art of the technical solution of the present disclosure may beembodied in the form of software. The computer software may be stored ina storage medium (such as ROM/RAM, diskette, or light disk) and includea plurality of instructions which are used to implement the method asdescribed in the various embodiments of the present disclosure by aterminal device (such as a television, a mobile phone, a computer, adevice, an air conditioner, or a network device, etc.).

The foregoing description merely portrays some illustrative embodimentsaccording to the disclosure and therefore is not intended to limit thepatentable scope of the disclosure. Any equivalent structural or flowtransformations that are made taking advantage of the specification andaccompanying drawings of the disclosure and any direct or indirectapplications thereof in other related technical fields shall all fall inthe scope of protection of the disclosure.

We claim:
 1. An image super-resolution method, wherein the methodcomprises: acquiring and amplifying an image to be processed, andextracting a scaling feature from the amplified image, to obtain a firstimage to be processed; sending the first image to be processed to aresidual network, for the residual network outputting a corrected secondimage to be processed; and restoring the second image to be processed togenerate a restored image, and outputting the restored image.
 2. Themethod according to claim 1, wherein the operation of acquiring andamplifying an image to be processed, and extracting a scaling featurefrom the amplified image, to obtain a first image to be processed,comprises: acquiring a low-resolution image to be processed, andpre-processing the image to be processed in a pre-processing convolutionlayer; and sending the pre-processed image to be processed to a scaleamplification module, amplifying the image to be processed based on apreset amplification scale, and extracting the scaling feature from theamplified image, to obtain the first image to be processed.
 3. Themethod according to claim 2, wherein the preset amplification scale isdefined as two times, three times, or four times.
 4. The methodaccording to claim 1, wherein the operation of sending the first imageto be processed to a residual network, for the residual networkoutputting a corrected second image to be processed, comprises: sendingthe first image to be processed to the residual network, processing thefirst image to be processed by a plurality of bottleneck residual unitsin the residual network, to generate the corrected second image to beprocessed; and sending the second image to be processed to a scalerestoring module.
 5. The method according to claim 4, wherein theresidual network comprises the plurality of bottleneck residual unitsand a convolution layer, and each bottleneck residual unit is connectedwith a weight normalization module.
 6. The method according to claim 5,wherein the bottleneck residual unit comprises three convolution layers,and an activation function layer is defined between each two adjacentconvolution layers, and the activation function is a PReLu function. 7.The method according to claim 6, wherein the activation functioncomprises a variable, and the value of the variable is obtained throughlearning from an upper network layer.
 8. The method according to claim1, wherein the operation of restoring the second image to be processedto generate a restored image, and outputting the restored image,comprises: on condition that a scale restoring module receives thesecond image to be processed, reducing the scale of the second image tobe processed based on a scale in a scale amplification module, togenerate the restored image; and outputting the restored image.
 9. Themethod according to claim 2, wherein the operation of sending the firstimage to be processed to a residual network, for the residual networkoutputting a corrected second image to be processed, comprises: sendingthe first image to be processed to the residual network, processing thefirst image to be processed by a plurality of bottleneck residual unitsin the residual network, to generate the corrected second image to beprocessed; and sending the second image to be processed to a scalerestoring module.
 10. An image super-resolution device, wherein thedevice comprises: a memory, a processor, and an image super-resolutionprogram stored on the memory and executable on the processor, theprogram, when executed by the processor, implements the followingoperations: acquiring and amplifying an image to be processed, andextracting a scaling feature from the amplified image, to obtain a firstimage to be processed; sending the first image to be processed to aresidual network, for the residual network outputting a corrected secondimage to be processed; and restoring the second image to be processed togenerate a restored image, and outputting the restored image.
 11. Thedevice according to claim 10, wherein the program, when executed by theprocessor, implements the following operations: acquiring alow-resolution image to be processed, and pre-processing the image to beprocessed in a pre-processing convolution layer; and sending thepre-processed image to be processed to a scale amplification module,amplifying the image to be processed based on a preset amplificationscale, and extracting the scaling feature from the amplified image, toobtain the first image to be processed.
 12. The device according toclaim 10, wherein the program, when executed by the processor,implements the following operations: sending the first image to beprocessed to the residual network, processing the first image to beprocessed by a plurality of bottleneck residual units in the residualnetwork, to generate the corrected second image to be processed; andsending the second image to be processed to a scale restoring module.13. A computer readable storage medium, wherein an imagesuper-resolution program is stored on the computer readable storagemedium, the program, when executed by a processor, implements thefollowing operations: acquiring and amplifying an image to be processed,and extracting a scaling feature from the amplified image, to obtain afirst image to be processed; sending the first image to be processed toa residual network, for the residual network outputting a correctedsecond image to be processed; and restoring the second image to beprocessed to generate a restored image, and outputting the restoredimage.
 14. The computer readable storage medium according to claim 13,wherein the program, when executed by the processor, implements thefollowing operations: acquiring a low-resolution image to be processed,and pre-processing the image to be processed in a pre-processingconvolution layer; and sending the pre-processed image to be processedto a scale amplification module, amplifying the image to be processedbased on a preset amplification scale, and extracting the scalingfeature from the amplified image, to obtain the first image to beprocessed.
 15. The computer readable storage medium according to claim13, wherein the program, when executed by the processor, implements thefollowing operations: sending the first image to be processed to theresidual network, processing the first image to be processed by aplurality of bottleneck residual units in the residual network, togenerate the corrected second image to be processed; and sending thesecond image to be processed to a scale restoring module.